The subject matter disclosed herein relates to a system and method capable of identifying the extent of a failure quickly in a messaging system.
In recent years, a huge amount of data (big data) is sent/received over a network nowadays, and the importance of a messaging technology for processing big data efficiently is rising. With a common messaging technology, processing of relaying a message (for example, sensor data) to a destination server is executed by using a messaging system that includes a message receiving server, a message sending server, and backup storage where messages are stored.
The messaging system executes processing described below in the message relaying processing.
First, the receiving server receives a message from a sending terminal, stores the received message in the backup storage, and then sends a response to the sending terminal. Thereafter, the sending server obtains the message from the backup storage, relays the obtained message to a destination server, and then deletes the message from the backup storage.
By following the processing steps described above, a response to the sending terminal can be sent as soon as the storing of the message in the backup storage is finished, and the response performance is accordingly improved. Storing a message in the backup storage also guarantees the permanence of the stored message.
In recent years, messaging systems capable of processing a large amount of data efficiently and quickly by utilizing in-memory Key Value Store (KVS) are becoming popular.
In-memory KVS is a form of data store where data that is a pair of a key and a value is stored in a volatile memory. With in-memory KVS, high scalability is accomplished and, because data is stored on the memory, high processing performance is achieved as well. In-memory KVS is also capable of preventing data loss, which is a concern in an in-memory environment, through the replication (duplication) of data between a plurality of servers.
In the following description, processing of storing data that is a pair of a key and a value in a volatile memory is referred to as storing, processing of reading a value that is associated with a key out of the volatile memory is referred to as retrieval, and processing of erasing data that is a pair of a key and a value from the volatile memory is referred to as deletion.
In a messaging system that utilizes the in-memory KVS described above, data is managed on the in-memory (volatile memory) for quick processing, which has a possibility of complete data loss in the event of multiple failures. Further, when a data loss occurs, identifying the lost data is difficult because the receiving server, the sending server, and an in-memory KVS all handle different aspects of message processing from one another. The resulting problem is that grasping the extent of a failure (how many users have been affected by the message loss, how many messages have been lost, and the like) is difficult.
When a data loss occurs in a messaging system and a sending failure due to the data loss cannot be notified to the sender side, the failure is generally treated as a service failure. Accordingly, when dealing with a data loss by restoring lost data, or by identifying the extent of the failure and sending an error response to the sending terminal, is not possible in a messaging system that utilizes in-memory KVS, there is a chance of service failure, which makes the messaging system an unstable system.
A known method of preventing a service failure by restoring data is a technology described in U.S. Pat. No. 4,159,517 A.
The technology described in U.S. Pat. No. 4,159,517 A is commonly known as Write Ahead Logging (WAL), which is a DB technology. Write Ahead Logging is a technology used in in-memory DBs, typically, Hbase, and involves recording the specifics of processing in a non-volatile storage medium as a log when a piece of data is written so that the piece of data is restored by re-executing the processing recorded in the log. In Write Ahead Logging, each time the receiving server and the sending server execute processing that is related to in-memory KVS, the in-memory KVS records in a non-volatile storage medium a log of the processing including the specifics of the stored data. In the case where a failure occurs in the in-memory KVS, the in-memory KVS is reactivated, and the processing recorded in the log is executed to recover to the state that precedes the failure. By following these steps, the system can return to the state prior to data loss and Write Ahead Logging is thus capable of preventing a service failure due to data loss.
A known method of preventing a service failure by ensuring that the extent of a failure can be identified is a technology described in JP 2004-227360 A.
JP 2004-227360 A includes the following description: “Session information, which is assigned to each execution of processing in log information held by a server, is recorded in a session information management table, and a session information association relation between different pieces of log information is recorded in a session information association table. The session information association table is searched recursively for a piece of session information that is specified by a user through a display target entering part, to thereby identify a series of pieces of session information related to processing that is of interest to the user. For each piece of session information out of the series of related pieces of session information, an associated piece of information is collected from the log information. The collected pieces of information are integrated by the date of recording of the log and formatted so that it is easy to understand the flow of processing. The formatted information is then presented to the user.”